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FOREWORD 


Thj  U.S.  Army  Resuarcti  Institute  for  the  Behavioral  and  Social  Sciences 
(ARI),  in  conjunction  with  PM-TRADE  and  Training  and  Doctrine  Command  (TRADOC), 
performs  research  and  development  that  will  lead  to  more  cost-effective  train¬ 
ing  devices.  To  this  end,  ARI  joined  with  PM-TRADE  and  TRADOC  in  a  Joint  Work¬ 
ing  Group  (JWG)  called  "AMTESS  AS  CBI."  This  report  was  initiated  by  JWG  to 
capitalize  on  and  climax  3  years  of  development  and  evaluation  of  AMTESS  proto¬ 
types.  It  does  so  by  extracting  lessons  learned  about  how  to  design  and  ac¬ 
quire  maintenance  training  devices. 

The  report  was  requested  during  the  October  1986  In  Training  Review  (IPR) 
with  PM-TRADE.  The  results  were  briefed  to  the  Commanders  of  PM-TRADE  and  ARI 
as  part  of  the  April  1987  IPR.  An  outcome  of  that  briefing  was  an  agreement 
by  the  Commanders  to  use  this  report  as  a  potential  basis  for  developing  de¬ 
cision  aids  for  device  specification  writers. 


EDGAR  M.  JOHNSON 
Technical  Director 
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ARMY  MAINTENANCE  TRAINING  AND  EVALUATION  SIMULATION  SYSTEM  (AMTESS): 
LESSONS  LEARNED 


EXECUTIVE  SUMMARY 


Requirement: 

Using  lessons  learned  from  the  AMTESS  Program  develop  and  test  guidelines 
on  how  to  acquire  training  devices. 

Procedure: 

The  AMTESS  program  produced  11  reports.  To  meet  the  objectives  stated 
above,  these  reports  were  systematically  searched  according  to  a  conceptual 
strategy  designed  to  yield  documentation  of  the  following: 

1 .  AMTESS  research  results  that  suggest  or  indicate  how  future  equip¬ 
ment  might  be  designed  more  effectively. 

2.  References  in  the  AMTESS  reports  to  sources  of  published  guidance 
on  how  to  design  training  equipment. 

3.  AMTESS  research  that  points  to  the  need  for  additional  research  data 
on  how  to  design  training  devices. 

4.  Suggestions  on  how  to  improve  acquisition  procedures  and  avoid  pit- 
falls  in  equipment  acquisition. 

A  categorization  scheme  that  organizes  the  extracted  documentation  into  six 
major  divisions  was  developed.  The  scheme  and  its  divisions  were  formatted 
to  provide  a  guide — a  list  of  "dos"  and  "dont's"  for  future  training  device 
acquisition  projects.  To  test  the  guide's  usefulness  it  was  tried  out  in  a 
reverse  engineering  mode  on  the  specifications  for  the  Bradley  Fighting  Ve¬ 
hicle  Maintenance  Trainer  (BFVMT).  The  guide  was  used  to  evaluate  the  qual¬ 
ity  of  specifications  for  the  hands-on  and  panel  components  of  the  trainer. 


Findings : 

Two  kinds  of  recommendations  emerged  from  the  AMTESS  experience.  One  is 
specific  and  quantitative:  For  example,  "CRT  screens  should  respond  in  .5 
seconds  to  learner  input. "  The  other  kind  of  recommendation  is  a  general  ap¬ 
plication  of  the  guide  to  the  Bradley  Trainer  Specifications,  indicating  that 
there  are  many  fundamental  problems  with  the  way  specs  are  now  written,  but 
also  many  opportunities  for  improvement,  even  with  the  current  state  of  knowl¬ 
edge  about  device  design  and  acquisition.  The  report  details  these  problems, 
as  well  as  specific  suggestions  for  solving  them. 


utilization  of  Findings: 


The  device  acquisition  guidelines  (DAG)  contained  in  this  report  can  help 
Training  and  Doctrine  Command  (TRADOC)  formulate  device  requirements  and  PM- 
TRADE  prepare  Request  for  Quotation  (RFQ)  specifications.  The  Device  Acquisi¬ 
tion  Guideline  (DAG)  can  point  to  categories  of  specification  that  need  to  be 
highlighted  as  well  "s  provide  some  of  the  specifications. 


ARMY  MAINTENANCE  TRAINING  AND  EVALUATION  SIMULATION  SYSTEM  (AMTESS): 
LESSONS  LEARNED 
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AEMY  MAINTENANCE  TRAINING  AND  EVALUATION  SIMULATION  SYSTEM  (AMTESS): 

LESSONS  LEARNED 


1.1  Overview 

a.  This  report  summarizes  the  final  effort  in  the  AMTESS^  Program. 

The  report  documents  "Lessons  Learned"  about  how  to  develop  specifications 
for  training  devices.  These  lessons  have  been  organized  as  a  set  of 
"Device  Acquisition  Guidelines".  In  Section  2  and  3  we  describe  how  the 
guidelines  were  constructed  and  applied  in  a  reverse  engineering  exercise 
to  the  specifications  for  the  Bradley  Maintenance  Trainer. 

b.  In  Sections  3  and  4,  we  recommend  how  the  current  practices  for 
specifying  device  characteristics  might  be  revised  to  better  account  for 
training  effectiveness  and  related  human  factors  issues.  In  retrospect, 
the  AMTESS  program  proved  to  be  an  invaluable  opportunity  to  illustrate 
how  device  acquisition  problems  could  be  reduced  through  better  specifica¬ 
tions  to  device  builders. 

1.2  Background:  AMTESS  Program 

a.  Excellent  sumnaries  of  the  AMTESS  program  and  what  inspired  its 
birth  are  presented  by  Hofer  (1981)  and  Dybas  (1983).  Additional  details 
are  provided  by  Evans  and  Mirabella  (1982) ,  and  in  the  AMTESS  evaluation 
reports  (Appendix  C) .  But,  briefly  the  program  goal  was  to  acquire  and 
test  alternative  generic,  maintenance  training  prototypes  and  then  use  the 
results  of  the  tests  to  produce  a  set  of  general  specifications  for 
acquiring  such  devices. 

b.  Prototypes  were  built  by  Grumman  and  by  Seville/feurtek.  Each  con¬ 
sisted  of  a  microcomputer,  one  or  more  two-dimensional  displays  and  a 
three-dimensional  mock-up  (e.g.  full-scale  diesel  engine,  starting  system 
components,  or  high  powered  Hawk  radar  transmitter) .  One  version  included 
a  cathode-ray  tube  (CRT)  with  motion  picture  and  sound  from  video-disc. 

The  second  included  a  CRT  with  no  video-disc,  but  with  computer-con- 
trolled,  35  nm  slide  projection. 

c.  We  tested  the  alternatives  by  measuring  how  much  skill  carried 
over  from  training  on  the  prototypes  to  actual  equipment  maintenance.  ARI 
and  its  contractors  (Science  ^plications,  Inc.,  and  Klein  Associates) 
conducted  the  tests  at  Aberdeen  (automotive  training)  and  at  Ft.  Bliss 
(electronic  training)  for  the  following  MOSs:  63B30,  63W10,  63H30,  and 
24C10. 


^  AMTESS:  Army  Maintenance  Training  and  Evaluation  Simulation  System 


d.  We  concluded  that  future  simulators  developed  from  the  proto¬ 
type-derived  specifications  could  be  cost-effective  as  part-trainers  to 
support  actual  equipment  training.  However,  as  designed,  they  should  not 
substitute 

e.  But  there  was  far  more  to  the  AMTESS  evaluation  reports  than  the 
performance  data  that  led  to  the  above  general  conclusions.  The  reports 
contained  a  wealth  of  information  about  acquisition  practices  and  design 
specifications  which  either  did  or  potentially  could  limit  the  training 
gffectiveness  of  the  devices.  Unfortunately  this  information  was  scat¬ 
tered  and  buried  in  hundreds  of  pages  of  mostly  narrative  text.  We  under¬ 
took  this  final  effort  in  the  AMTESS  Program  to  pull  together  and  organize 
in  a  usable  way  seme  specific  ideas  for  what  to  include  in  device  specs. 

f .  In  this  report  we  analyze  the  past  reports  and  the  "lessons 
learned"  concerning  the  above  acquisitions  practices  and  design  specifica¬ 
tions.  Our  "lessons  learned"  flag  the  features  of  the  acquisitions  proc¬ 
ess  that  are  most  likely  to  procure  inferior  devices.  The  effort 
resulting  Devices  Acquisition  Guidelines  should  be  used  by  the  specifica¬ 
tion  writer  to  guide  and  check  his  specifications  preparation.  Thus  the 
writer  will  avoid  some  of  the  most  frequent  mistakes  in  specifications 
preparation.  These  guidelines  should  not  be  taken  lightly.  The  problems 
we  have  identified  were  produced  by  experts  who  were  responding  to  a 
statanent  of  work  that  already  emphasized  good  specification  preparation. 

1.3  Background:  Scientific 

a.  The  broader  scientific  problem  of  how  to  translate  human  factors 
and  instructional  design  data  and  principles  into  a  form  which  training 
equipment  designers  can  and  will  use  is  an  old  and  persistent  "chestnut" 
in  applied  psychology.  Meister  studied  this  problem  20  years  ago.  He 
provided  a  blueprint  for  a  solution,  which  has  has  been  almost  entirely 
ignored. 

b.  Meister  et  al.  (1965a,  1965b,  1966a,  1966b,  1966c,  1967,  1968, 
1969a,  1969b,  1971)  examined  the  behavior  of  equipment  designers  at 
Bunker-Ramo,  Douglas  Aircraft,  Marquardt,  and  Raytheon  Corporations  as 
they  designed  Air  Force,  aero-space,  and  Navy  equipment  under  a  variety  of 
controlled  experimental  conditions.  His  purpose  was  to  determine  the 
normal  use  of  human  factors  information,  the  normal  quality  of  human  fac¬ 
tors  design,  and  the  condtions  under  which  equipment  design  would  be  im¬ 
proved.  He  found  that  design  engineers  didn't  normally  use  human  factors 
principles,  didn't  have  human  factors  references,  and  couldn't  use  human 
factors  references  when  they  were  provided.  They  would  apply  good  human 
factors  and  design  excellent  equipment  only  when  the  human  engineering 
information  was  provided  in  the  style  that  they  were  accustomed  to,  they 
were  forced  to  use  it  by  the  SOW,  and  the  human  factors  constraints  in 
terms  of  numbers  and  types  of  personnel  were  specified  in  the  SOW. 


c.  Meister  concluded  that  the  only  way  to  obtain  operable  and 
maintainable  equipment  is  to  provide  detailed  functional  specifications  in 
the  SOW.  These  must  provide  and  highlight  the  human  factors  constraints 
on  the  system  design. 

d.  William  Askren,  who  sponsored  some  of  Meister 's  research,  continued 
it  at  Wright-Patterson  AFB.  He  stated  in  a  recent  telephone  conversation 
that  Meister 's  research  was  applied,  in  part,  by  the  Air  Force,  and  that 
it  was  the  genesis  of  the  Navy  HAREWAN  effort  which,  in  turn,  led  to  the 
Army  MIST  and  MANPRINT  programs.  The  AMTESS  "lessons  learned"  demonstrate 
that  his  principles  for  good  equipment  design  are  as  valid  now  for  train¬ 
ing  devices  as  they  were  for  the  actual  equipment  then. 

e.  Yet,  these  principles  have  not  been  applied  with  demonstrable  suc¬ 
cess  to  the  design  and  acquisition  of  Army  training  devices.  PM-TRADE, 
through  its  support  contractor.  Science  Applications  International  Corpo¬ 
ration  is  beginning  to  apply  the  six  categories  of  MANPRINT  analyses  to 
concept  formulation.  And,  in  the  supporting  OSBAT  effort,  ARI  is  develop¬ 
ing  algorithms  to  optimize  the  instructional  design  of  training  devices 
(Sticha  et  al,  1986) .  How  well  these  approaches  will  translate  into  pre¬ 
cise,  comprehensive  specifications  and  cost-effective  training  remains  to 
be  determined.  But  it  should  be  emphasized  that  the  Bradley  specifica¬ 
tions  which  we  analyzed  were  developed  at  least  partially  under  the  new 
MANPRINT  rules.  One  outcome  of  that  analysis,  incidently,  was  to  high¬ 
light  this  translation  problem  and  suggest  that  it  may  remain  inspite  of 
the  excellent  efforts  of  MANPRINT  and  OSBATS.  The  PM-TRADE  contractor 
agreed  that  "there  is  a  requirement  for  a  procedure  to  transition  training 
requirements  into  specification  languages"  (Elam,  1987). 

2.0  METHOD 

2.1  Development  of  Device  Acquisition  Guidelines  (DAG) 

a.  The  objective  of  this  exercise  was  to  systanatically  collect  the 
research-based  guidance  in  the  AMTESS  reports  that  bears  most  directly  on 
the  production  of  engineering  specifications  for  the  manufacture  of  main¬ 
tenance  training  devices. 

b.  To  develop  the  contents  of  the  Device  Acquisition  Guide  (DAG)  all 
relevant  AMTESS  reports  were  ccxnpiled  and  analyzed  by  training  experts 
familiar  with  the  device  acquisition  process  and  with  the  AMTESS  Program. 
They  were  instructed  to  note  and  reference  occurrences  of  the  following; 

(1)  AMTESS  research  results  which  suggest  or  indicate  how  future 
equipment  might  be  designed  more  effectively. 

(2)  References  in  the  AMTESS  reports  to  sources  of  published 
guidance  on  how  to  design  training  equipment. 

(3)  AMTESS  research  which  points  to  the  need  for  addi tonal 
research  data  on  how  to  design  training  devices. 


(4)  Suggestions  on  how  to  improve  acxjuisition  procedures  and 
avoid  pitfalls  in  equipment  acquisition. 

c.  Each  of  four  experts  incl^)endently  read  the  AMTESS  reports  and 
extracted  material  judged  to  fit  the  objective.  We  then  edited  their 
respective  products  to  eliminate  duplications,  and  we  consolidated  the 
guidelines  under  eighteen  subcategories  of  training  device  features  and 
capabilities  that  could  be  specified  in  statanents  of  work  to  procure 
devices. 

2.2  Reverse  Engineering  of  Bradley  Trainer  Specs 

a.  We  applied  the  DAG  to  the  specifications:  for  the  Bradley  Fight¬ 
ing  Vehicle  Maintenance  Trainer  (BFVMT).  PM-TRADE  selected  these  specifi¬ 
cations  as  the  target  application  of  the  DAG.  Our  aim  was  to  determine 
whether  and  how  well  those  specifications  treated  training  effectiveness 
requirements  which  were  short-falls  in  the  AMTESS  acquisition.  For  this 
exercise,  training  effectiveness  included  human  factors,  instructional 
design,  and  integration  of  the  Bradley  trainer  into  a  program  of  instruc¬ 
tion.  This  broad  definition  is  critical  since  good  instructional  design 
is  necessary  but  not  sufficient. 

b.  Bradley  Specs:  We  reverse  engineered  two  specifications  (Naval 
Training  Systems  Center  1986a,  1986b):  The  first  specified  four  panel 
trainers  with  associated  media  (e.g.  projection  systan,  student  station, 
instructor  station) .  The  second  defined  requirements  for  the  Hands-on- 
trainer,  Device  17-78. 

2.3  Assumptions.  The  validity  and  usefulness  of  our  results  and  conclu¬ 
sions  depends  on  two  assunptions: 

a.  Reasonableness  of  the  AMTESS  program  as  a  test  bed  for  cataloging 
Amy  device  acquisition  problems.  The  AMTESS  program  is  more  than  rea¬ 
sonable,  because  an  extraordinary  amount  of  time  and  money  were  spent 
developing  training  effectiveness  requirements,  proportionately  far  more 
than  is  ever  allocated  to  a  training  device  program  even  under  the 
MANPRINT  mandate.  Five  conpanies,  independently  conducted  concept  formu¬ 
lations  for  the  devices  and  the  best  two  were  selected  to  further  develc^ 
and  build  the  AMTESS  prototypes. 

b.  Representativeness  of  the  Bradley  Specs.  These  specs  were  provided 
by  PM-TRADE  for  this  project.  We  have  assumed  that  they  are  typical  for 
Amy  training  device  acquisitions.  Since  this  was  an  exploratory  effort 
with  very  limited  resources  we  could  not  analyze  a  large  sample  of  device 
specifications  to  verify  the  assumption.  A  follow-up  effort  should  do  so. 

2.4  Study  Questions 

a.  What  lessons  can  we  extract  from  the  AMTESS  Program? 

b.  What  do  those  lessons  tell  us  about  the  quality  of  current  training 
device  specifications? 


c.  How  can  we  improve  those  specifications,  using  state-of-the  art 
knowledge  and  technology? 

3.0  RESULTS 

3.1  Lessons  Learned  From  AMTESS 

a.  Device  Acquisition  Guidelines  (DAG).  Appendix  A  presents  the  DAG 
which  was  derived  from  the  analysis  of  AMTESS  reports.  We  gleaned  state¬ 
ments,  conclusions,  data,  observations  and  do's  and  don'ts  from  those 
reports  and  then  clustered  them  into  the  most  meaningful  categories  we 
could  think  of.  The  DAG  is,  however,  more  an  experience-based  taxonomy  of 
what  can  go  wrong  in  an  acquisition.  It  flags  topics  the  specification 
writer  should  pay  special  attention  to  in  addressing  training  effective¬ 
ness  requirements. 

b.  The  DAG  categories: 

(1)  Man-Machine  Conmunication.  This  category  includes  all  those 
design  features  which  affect  how  well  the  operator  and  the  device  "talk" 
to  and  react  to  each  other.  Before  the  operator  can  learn  anything  he 
must  receive  information  through  his  eyes,  ears,  muscle  sensors,  and  skin, 
comprehend  the  meaning  of  the  information,  and  then  talk  back  to  the 
machine.  This  may  sound  obvious.  Yet,  one  of  the  AMTESS  devices  was 
judged  to  require  reading  abilities  beyond  those  of  the  trainees. 

(2)  Hardware:  A  number  of  somewhat  miscellaneous  design  charac¬ 
teristics  do  not  fit  neatly  into  the  "communication"  or  instructional 
design  categories,  yet  can  affect  the  performance  directly  or  indirectly 
of  device  users.  These  have  been  subdivided  into  hardware  design,  safety/ 
hazards,  and  design  for  maintainability. 

(3)  Acquisition  Management.  With  this  category  we  shift  away 
from  design  specifications  per  se  to  some  do's  and  don'ts  on  how  to 
manage  the  device  acquisition.  Again,  as  with  the  previous  categories 
these  recommendations  are  really  AMTESS  problems  re-stated  as  guidelines. 

(4)  Human  Factors  and  Software  Design.  Much  can  be  done  to 
misdesign  computer  software  to  impair  the  "human  factor".  Numerous  exam¬ 
ples  were  uncovered  in  the  AMTESS  evaluation  and  these  have  been  sub-di¬ 
vided  into  problems  affecting  the  trainee  station  and  those  affecting  the 
instructor  station. 

(5)  Instructional  Delivery.  This  category  recognizes  that 
simulators  are  more  than  just  3-dimensional  mock-ups.  They  are  first  and 
foremost  teaching  devices  and  as  such  need  to  include  features  which  help 
instruct  and  assess  student  performance.  And,  their  efficient  use 
requires  that  they  be  properly  integrated  into  the  relevant  POI  along  with 
other  media  such  as  2-D  part  trainers.  Accordingly,  we  have  sub-divided 
this  category  into  instructional  features/principles,  and  student  record 


(6)  Fidelity.  This  could  well  be  a  third  sub-division  under 
instructional  delivery,  but  we've  isolated  it  as  a  major  category  because 
device  developers  and  builders  have  a  special  concern  for  it.  Fidelity 
here  refers  to  how  much  a  simulation  system  resanbles  the  appearance  and 
behavior  of  the  target  weapon  systan.  The  issue  of  part  vs  whole  training 
is  included  within  this  category. 

3.2  What  is  the  quality  of  current  training  device  specifications? 

a.  How  well  do  the  Bradley  specs  address  the  problems  cataloged  in  the 
DAG?  We  answered  this  question  by  "reverse  engineering"  the  specifica¬ 
tions  for  the  Bradley  Maintenance  Hands  on  Trainer  (HOT)  and  Panel 
Trainers.  In  the  first  of  two  procedures  we  enumerated  the  DAG  categories 
and  sub  categories  which  were  treated,  not  treated,  or  violated  in  the 
Bradley  specs.  The  results  of  this  enumeration  are  shown  in  Tables  1-3, 
Appendix  B.  Table  1  sumnnarizes  the  item- treatment  by  specification. 

Tables  2  and  3  present  a  more  detailed  brealcdown  by  DAG  category.  The 
data  show  that  about  90%  of  the  DAG  items  (i.e.  AMTESS  problems)  were  not 
addressed  in  the  Bradley  specs  and  therefore  constitute  potential  probl^ns 
in  the  Bradley  trainers. 

b.  What  is  the  quality  of  the  content  and  structure  of  the  Bradley 
specs?  We  read  and  analyzed  the  specs  in  detail  with  the  help  of  the  DAG. 
We  arrived  at  five  major  findings. 

(1)  The  word  "training"  is  barely  mentioned  in  either  specifica¬ 
tion.  Out  of  164  numbered  paragraphs  in  the  panel  specs,  only  12  dealt 
with  training  issues.  Out  of  145  numbered  paragraphs  in  the  HOT  specs, 
only  7  dealt  with  training  issues.  The  remaining  narrative  paragraphs 
(Pages  1  -30  in  the  HOT  specs,  and  1-35  in  the  panel  specs)  in  both  docu¬ 
ments  dealt  with  engineering  specs  for  mock-ups,  as  opposed  to  "trainers". 

(2)  The  specs  contain  no  indication  about  how  the  devices  are  to 
be  integrated  into  a  program  of  instruction,  e.g.  where  and  vAien  are  they 
to  be  used?  T^hat  training  philosophy  and  strategy  should  they  si;qpport? 
^Failure  to  address  this  issue  and  its  implied  questions  proved  to  be  a 
major  problem  for  the  AMTESS  program.  Answers  to  these  questions  can  and 
should  influence  the  design  of  the  training  hardware  and/or  software. 


^  Elam  wrote  us  that  "the  strategy  of  device  use  should  then  be  part  of 
the  TDR  which  would  assist  us  in  determining  (1)  how  many  student  devices 
will  be  required  and  (2)  what  training  is  to  be  acccxiplished  by  the  device 
in  relation  to  the  task.  Seldom  accomplished". 


^  Elam  stated  that  "TRADOC  would  be  supporting  this  by  the  development  of 
the  course  curriculuir/syllabus  which  defines  the  strategy  to  be  used  with 
the  device"  (op.cit.). 


(3)  The  specs,  particularly  those  for  the  panel  trainers,  have 
serious  and  basic  problems  in  logical  organization.  As  a  result,  the  few 
references  to  training  effectiveness  requirements  are  difficult  to  track. 
The  general  problen  is  that  paragraphs  are  not  organized  into  topic  sen¬ 
tences  and  supporting  explanations.  Instead,  related  items  of  information 
are  often  spread  across  several  paragraphs.  For  example  on  Page  7,  Para¬ 
graph  3. 2. 1.2  of  the  BFV  Panel  Trainer  Specifications,  the  author  states 
"The  trainee  shall  be  required  to  perform  all  troubleshooting  tasks  in 
order  and  within  a  time  standard".  He  explains  this  sentence  four  para¬ 
graphs  later  in  Paragraph  3.2.3. 

(4)  Many  terms  and  requirenents  are  inadequately  defined  and 
therefore  subject  to  being  ignored  or  misinterpreted: 

(a)  For  example,  on  Page  7  of  the  BFV  Panel  Trainer  Specs, 
(ParagrafAi  3. 2. 1.1)  the  author  states:  "The  training  devices  shall  enable 
trainees  to  become  proficient  ..."  What  is  the  meaning  of  this  require¬ 
ment  and  what  implication  does  that  meaning  have  for  the  design  of  a 
trainer  as  opposed  to  the  design  of  a  simulator?  And,  how  will  the  govern¬ 
ment  know  if  this  requirenent  has  been  met?  It's  well  within  the  state  of 
the  art  to  specify  a  testing  procedure  and  criterion  for  defining  what  is 
meant  by  "proficient."  ^ 

(b)  A  second  example,  is  the  statement  on  Page  7,  Paragraph 
3. 2. 1.1:  "The  BFVS  Panel  Trainers  shall  provide  the  capabilities  for  the 
instructor  to  demonstrate  troubleshooting  procedures  ...  to  a  variable 
class  size  in  a  classroom  environment."  Both  "capabilities"  and  "class 
size"  are  undefined.  Capabilities  for  example,  should  include  panel  dis¬ 
plays  which  can  be  seen  and  understood  five  feet  back  and  60  degrees  off 
the  perpendicular  which  is  where  several  of  the  trainees  are  going  to  be 
when  this  panel  is  demonstrated  by  the  instructor.  This  may  require  dis¬ 
plays  which  are  artificially  large.  Here  is  a  case  where  it  might  be 
necessary  to  distort  fidelity  of  displays  in  order  to  use  the  device 
effectively  for  demonstrations. 

(5)  Citing  Mil  Standard  1472  Doesn't  Work.  Citation  of  1472  in 
the  AMTESS  specs  did  not  prevent  extensive  training  effectiveness  problems 
in  the  AMTESS  devices,  nor  is  it  likely  to  do  so  in  the  Bradley  trainer. 

A  series  of  studies  1:^  David  Meister  beginning  in  1967  indicated  that 
equipment  designers  ignore  human  factors  standards  or  don't  know  what  to 
do  with  them.  Meister  suggested  some  alternatives  to  blanket  citation  of 
Mil  standards.  His  suggestions  were  ignored.  As  a  result,  we  have  not 
made  much  progress  in  20  years  in  helping  engineers  to  design  the  hunan 
factor  into  training  equipment. 


^  Elam  suggested  including  Measures  of  Training  Effectiveness  (NOTE)  such 
as  "%  passing  final  test  on  device  and  %  passing  on  first  trial  in  actual 
equipment",  (op.  cit.) 


3.3  What  Can  We  Do  to  Inprove  Training  Device  Specifications  Using 

State  of  the  Art  Knowledge? 

We  certainly  don't  know  enough  about  how  to  design  training  equipment. 
But  then  we  didn't  know  enough  20  years  ago  and  probably  won't  know  enough 
20  years  hence.  Nonetheless  we  can  do  a  better  job  of  using  what  we  do 
know  or  may  know  soon  because  of  current  research. 

a.  We  can  begin  by  solving  seme  of  the  general  problems  cited  above. 
Those  problems  are  not  beyond  the  state  of  the  art,  but  they  do  call  for 
more  skills  than  the  current  specification  writers  have,  e.g.  instruct i(x\- 
al  design,  human  factors,  training  effectiveness  analysis,  and  technical 
writing,  in  addition  to  engineering.  We  don't  expect  the  anesthesiologist 
to  double  as  the  brain  surgeon.  Yet,  we  ask  the  specification  writer  to 
do  the  job  of  five  specialists.  However,  as  an  alternative  to  hiring  five 
specialists  for  each  device  acquisition,  we  can  develop  decisicwi  and  job 
aids.  We  need  aids  that  will  help  the  acquisition  engineer  organize  his 
ideas  cind  use  v^tever  training  effectiveness  data  and  guidelines  we  can 
muster. 

b.  We  can  also  use  the  lessons  learned  from  AMTESS  or  any  other  acqui¬ 
sition  e^qperiences  to  identify  specific  device  acquisition  prdDlems  and 
then  address  those  problans  explicitly  in  the  bocty  of  the  specifications. 
This  is  v^t  Meister  recommended  20  years  ago.  Examples  of  how  this  might 
be  done  are  presented  in  Table  1.0.  The  table  presents  the  AMTESS  defi¬ 
ciency  eissociated  with  seven  of  the  DAG  subcategories  as  well  as  the  re¬ 
sult  of  the  analysis  of  the  BFVMT  specifications.  This  portion  of  the 
table  demonstrates  that  the  DAG  is  relevant  to  recent  trainii^  device 
procurements  because  the  BFVWT  specifications  failed  to  specify  device 
characteristics  that  proved  troublesome  in  the  AMTESS  devices. 


TABLE  1.0 


Examples  of  How  to  Write  More  Precise  Training-Related  Specifications 


DAG  Category 

AMTESS  Deficiency 

Bradley  Deficiency 

Suggested  Specs 

AUDIO 

INFO  RATE 

EXCESSIVE 

SPEECH  RATE 

NOT  ADDRESSED 

80  TO  120  WPM 

DISPLAY 

DESIGN 

READING  LEVEL 

TOO  HIGH 

NOT  ADDRESSED 

USE  FOG  OR  FORCAST 
E.G.  63MOS  =  8TH  GRA 

controla>isplay 

INTEGRAT. 

COMPUTER  RESPONSE 
TIME  EXCESS. 

INCORRECT 

INCOMPLETE 

MISLEADING 

CRT:  .5SECS. 
PANEL:  HI  FI 

SIMULATION 

FIDEELITY 

TOO  MUCH  HI  FI  FOR 
TROUBLE  SHOOTING 

80  HARCWIRED 
EXCERCISES 

HARDWIRE  20%  OF 
REPEATED  EXERCISES 

SOFTWARE 

DESIGN 

RESTARTS  FROM 
BEGINNING 

NOT  ADDRESSED 

RESTART  AT  STOP 
POINTS 

STUDENT  RECORD 
KEEPING 

PRINTER  OUT 

PUT;  ILLEGIBLE 
UNREADABLE  CODE 

QUALITY,  FORMAT 
CONTENT  NOT  SPEC'D 

LETTER  QUAL  10  POINT 
8TH  GRADE  ENGLISH 

STUDENT  RECORD 

INADEQUATE 
SUMMARY  DATA 

NOT  ADDRESSED 

SPECIFY  SUMMARIES 
E.G.  BY  TASK  BY  CLASS 

Coirmentary  on  Table  1.  Several  items  in  the  table  need  some  explanation 


1.  "Use  FOG  or  Forecast"  Readability  in  the  Amy  has  taken  on  new  iirpor- 
tance  with  the  recent  publication  of  new  DA  regulations  on  clear  writing 
(AR  600-70).  And,  it  is  certainly  critical  in  design  of  training  device 
dis-plays.  ARI's  "Guidebook  for  the  Developnent  of  Amy  Training  Litera¬ 
ture"  explains  what  "readability"  means  and  how  to  estimate  it.  The 
guidebook  also  lists  examples  of  reading  levels  for  a  sample  of  MOSs. 

2.  "CRT:  .5  Secs.,  Panel:  Hi  Fi."  What  this  cryptic  message  says  is 
that  a  training  device  may  have  different  conputerized  con^nents  and  that 
different  computer  response  times  may  be  called  for.  In  the  Bradley,  the 
student  CRT  display  should  respond  to  student  actions  in  about  .5  seconds, 
but  the  displays  on  the  panels  should  mimick  the  response  times  of  the 
actual  equipment  being  simulated. 

3.  "Hardwire  20%  of  repeated  exercises."  The  Bradley  specs  tell  the 
device  contractor  to  hard  wire  80  trouble-shooting  exercises.  For  many 
reasons,  it  would  be  more  cost-effective  to  hard-wire  just  a  few  of  these, 
e.g.  20  or  25%,  and  then  present  the  rest  on  a  2-D  device,  e.g.  inter¬ 
active  video  disc.  Such  a  mixture  of  media  would  allow  tte  trainee  to 
"feel"  the  actual  hardware  but  do  most  of  his  training  on  the  more  flexi¬ 
ble  and  available  student  station. 

4.0  CONCLUSIONS 

4.1  The  collection  of  lessons  learned  from  the  AMTESS  Program  provide  a 
useful  and  usable  guide  to  likely  deficiencies  in  training  device  acquisi¬ 
tions. 

4.2  Training  device  specifications  as  represented  by  those  for  the  Bradl^ 
Maintenance  Trainer  may  be  improved  significantly  liy: 

a.  Organizing  topics  and  suj^rting  explanations  with  clear,  deliber¬ 
ate,  and  demonstrable  logic. 

b.  Defining  training  effectiveness  terms  and  requirements  explicitly 
and  showing  how  they  might  affect  the  design  of  the  training  device. 

c.  Explaining  how  the  device  is  to  be  used  in  the  program  of  instruc¬ 
tion,  e.g.  indicating  what  training  philosophy  and  strategy  need  to  be 
supported  by  the  device. 

d.  Incorporating  into  the  body  of  the  specification  those  parts  of 
mili  tary  standards  that  are  most  relevant  to  the  particular  acquisition 
rather  than  relying  on  blanket  citations  or  even  abstract  checklists  of 
those  standards. 


4.3  It  is  possible  to  significantly  improve  the  quality  of  training 
device  specification  with  currently  availeible  knowledge. 

4.4  Examples  of  what  is  wrong  with  current  specifications  and  how  to 
improve  them  were  provided  in  the  foregoing  report. 

4.5  Job  and/or  decision  aids  are  needed  to  help  the  training  device 
specification  writer  function  ccn5>etently,  since  he  is  now  required  to 
perform  the  functions  of  five  different  specialists. 


Appendix  A 

DEVICE  ACQUISITION  GUIDELINES 
Using  the  Device  Acquisition  Guidelines. 

A.  What  They  Are 

1.  The  Device  Acquisition  Guidelines  (DAG)  are  lessons  learned 
from  problems  faced  in  acquiring  four  prototype  maintenance  trainers  in 
the  Arn^  Maintenance  cind  Evaluation  Simulation  System  (A^f^ESS)  acquisi¬ 
tion.  More  specifically  the  DAG  consists  of  quotes  and  paraphrases  taken 
from  a  number  of  r^xirts  investigating  the  problems  of  that  acquisition 
as  well  as  the  device  evaluation  results.  They  can  be  used  to  alert 
personnel  in  the  training  device  acquisition  process  to  ccmnon  acquisi¬ 
tion  problems. 

2.  The  DAG  is  not  meant  to  be  exhaustive,  but  it  does  identify 
and  help  solve  some  of  the  more  coomon  and  critical  human  factors  and 
training  design  acquisition  issues.  The  guidelines  are  based  on  good 
administrative,  training  and  human  factors  procedures. 

3.  The  answer  to  the  question  "Why?"  for  any  guideline  is  that 
it  was  violated  and  the  violation  was  related  to  a  problem  with  at  least 
one  training  device  acquisition.  Each  guideline  is  the  result  of  a  spe¬ 
cific  problem  in  the  acquisition  of  an  AMTESS  training  device. 

B.  AMTESS  Preferences  in  the  DAG: 

Each  guideline  is  followed  by  a  citation  in  the  form  (n)  p. 
nn.n.  The  number  in  parentheses  is  the  number  of  the  reference  in  appen¬ 
dix  C  from  which  the  guideline  was  extracted.  The  nn.n  specifies  the 
page  and  paragraf^i  containing  the  guideline  so  that  the  interested  user 
can  determine  the  exact  conditions  that  produced  the  guideline.  Some¬ 
times  the  reference  occurs  in  the  middle  of  a  guideline  so  that  it  can  be 
next  to  the  specific  phrase  with  v^ich  it  is  associated.  When  this 
occurs,  the  reference  is  separated  from  the  text  with  square  brackets. 


C.  Information  that  was  not  derived  from  AMTESS 

Sometimes  we  emplified  the  AMTESS  guidance  with  administrative/ 
training/human  factors  expertise  that  was  not  derived  directly  from  the 
AMTESS  studies.  This  information  is  enclosed  in  square  brackets  eind  is 
not  followed  with  a  reference.  Guidelines  1.6.1,  I.B.S.b,  and  I.D.l  pro¬ 
vide  examples. 

D.  How  to  use  the  DAG 

We  believe  that  the  best  way  to  use  the  DAG  is  as  a  review 
device. 

1.  Prepare  an  outline  of  the  desired  document. 

Check  the  DAG  to  find  topics  that  should  be  added  to 
the  outline. 

Reorganize  the  outline  to  bring  related  topics 
together . 

2.  Write  the  document. 

Fill  in  the  outline. 

Use  the  DAG  to  provide  the  required  specific  informa¬ 
tion. 

3.  Review  the  product. 

Compare  each  section  of  the  product  against  the  rele¬ 
vant  sections  of  the  DAG. 
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Include  any  missing  information. 

Perform  normal  review. 

This  procedure  should  enable  you  to  avoid  many  of  the  usual  pitfalls  in 
training  device  acquisition. 

E.  DAG  Categories 

The  DAG  is  arranged  with  seven  main  categories.  The  arrangement 
is  shown  below: 

I.  HLWAN  FACTORS 

A.  Audio  Presentation. 

B.  ControlsA>isplay  Integration. 

C.  Ease  of  use. 

D.  Visual  displays  and  textual  materials. 

E.  Workspace  Design. 

II.  HARDWARE 

A.  Design. 

B.  Safety  and  Hazards. 

C.  Maintenance. 

III.  ACQUISITION  MANAGEMENT 

A.  Front  End  Analysis  (FEA) . 

B.  Evaluation  Requirements 

C.  Training  Device  RFP/SCW: 

D.  Design  and  device  acceptance  procedures: 

IV.  HUMAN  FACTORS  and  SOFTWARE  DESIGN 

A.  Instructor  Station. 

B.  Student  Station. 
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V.  INSTRUCTIONAL  DELIVERY 


A.  Instructional  Features  and  Principles. 

B.  Student  record  keeping. 

VI.  FIDELITY 

A.  Basic  considerations. 

B.  Two  vs.  Three  Dimensional  Considerations. 

VII.  NOTES 


I.  HUMAN  FACTORS 


A.  Audio  Presentation. 

1.  By  analogy  with  video  information  presentation,  audio  presenta¬ 
tions  should  be  at  cin  appropriate  rate  and  volume  of  information  so  as  to 
not  overload  the  device  user  or  compete  unduly  with  other  task  training 
stimuli.  (1)  p.  50.1.  Rule  of  thumb:  80  to  120  words  per  minute. 

2.  Identical  audio  signals,  such  as  a  "beep"  tone  should  not  be 
used  to  signal  different  task  training  conditions  (e.g.,  the  occurrence 
of  an  error  vs  a  ready  response).  (1)  p.  27.5,  51.3 

3.  Realistic  simulation  of  sound  effects  should  be  provided  when 
identified  as  important  cues  in  the  FEA.  (3)  p.  34.7 

4.  The  presence  of  alerting  or  warning  displays  will  improve  user 
detection  of  conditions.  (1)  p.  24.3 

5.  A  signal  (audio  or  visual)  will  not  have  multiple  meanings. 

(1)  Ep.  27.5,  51.3 

B.  Visual  displays  and  textual  materials. 

1.  Visual  displays  should  incorporate  an  appropriate  volume  of 
information  per  frame  or  display  so  as  to  not  overload  the  user  with  com¬ 
peting  information.  (1)  pp.  49.7,  50.2  [Rule  of  thumb:  5  lines  of  text 
5  words  per  line.] 

2.  For  visual  displays,  requiring  the  student  to  mentally 
trans-late  displayed  information  should  be  avoided.  Rather,  displayed 
information  should  be  presented  as  follows: 

(a)  Display  only  relevant  info, 

(b)  Info  is  as  precise  as  needed. 


(c)  Displayed  info  is  inmediately  usable, 

(d)  The  information  ranains  displayed  long  enough  to  be 
detected.  (1)  p.  24.2-8 

3.  The  language  conprehension  level  (readability)  employed  in 
visual  displays  should  be  at  a  level  appropriate  to  student  reading  com¬ 
prehension  generally  an  average  high  school  reading  level  for  military 
enlisted  personnel.  (1)  p.  43.4 

(a)  Apply  the  fog  index  or  other  readability  test  to  all  tex¬ 
tual  materials.  The  maximum  acceptable  reading  level  is  the  minimum 
reading  level  for  the  MOS's  being  considered  for  training,  as  indicated 
by  the  school  outpoint.  (1)  p.  43.4 

(b)  [See  ARI  GUIDEBOOK  FOR  THE  DEVELOPMENT  OF  ARMY  TRAINING 
LITERATURE  on  how  to  measure  readability  and  for  examples  of  average  read¬ 
ing  abilities  in  Army  personnel.] 

4.  The  presence  of  alerting  or  warning  displays  will  improve  user 
detection  of  conditions.  (1)  p.  24.3 

5.  A  signal  (audio  or  visual)  will  not  have  multiple  meanings. 

(1)  p.  27.5 

6.  The  system  status  mode  or  condition  should  always  be  clearly 
displayed  for  students  and  instructors.  (1)  p.  24.2 

7.  All  labels  will  be  legible,  visible  from  the  student  location, 
permanent,  and  located  per  MIL  SPEC  1472C,  unless  the  device  is  a  simula¬ 
tion  which  requires  high  physical  fidelity.  (1)  p.  27.7,  p.  51.4 

8.  Panel  displays  should  not  be  ambiguous.  (1)  p.  51.4 

9.  Text  materials  presented  on  the  student  displays  will  be  coor¬ 
dinated  with  the  training  being  administered  and  with  the  technical  manu¬ 
als.  (3)  p.  33.2 

C.  Controls/Display  Integration. 

1.  Control/Display  relationship  should  be  immediately  a^^rent  to 
device  users.  (1)  p.  24.1 

2.  Device  response  features  and  student  input  controls  should  emu¬ 
late  the  performance  sensitivi^  of  counterpart  real  world  features  (e.g., 
the  training  device  features  should  not  be  unrealistically  sensitive) . 

(9)  p.  83.2&45 


3.  Device  computer  programs  and  user  input/output  features  should 
respond  quickly  enough  and  provide  enough  detailed  and  precise  information 
to  accomplish  the  system's  purpose  without  stressing  or  discouraging  the 
user  or  degrading  user  attention.  (1)  p.  24.10.  The  following  are  guide¬ 
lines  to  achieve  the  above: 

(a)  [In  normal  operation  the  system  will  provide  prompt 
acknowledgment  of  student  input,  0.4  to  2.0  seconds.] 

(b)  [Non-psychomotor  trainers  will  not  provide  feedback  in 
less  than  0.1  second.] 

(c)  (Psychomotor  e.g.  3D  trainers  will  replicate  the  response 
characteristics  of  the  actual  equipment.  Provision  may  be  made  for  varia¬ 
ble  system  response  to  provide  for  slower  or  faster  system  response  char¬ 
acteristics  if  these  would  improve  training  (i.e  shaping  the  behavior  is 
required) . ] 


(d)  [Multisense  psychcmotor  simulation  trainers  (e.g.  vision 
and  motion)  must  have  the  cues  to  the  different  senses  properly  synchro¬ 
nized.  This  is  more  important  than  replicating  the  response  characteris¬ 
tics  for  a  single  sense  modality  exactly.  If  the  cues  to  the  different 
senses  cannot  be  synchronized,  then  the  less  important  sensory  modality 
should  be  disabled.] 

4.  Device  response  time,  in  the  case  of  reinforcement  and  feed¬ 
back,  should  reinforce  or  signal  the  student  in  a  timely  manner  without 
undue  delay.  (9)  p.  83.8&10 

D.  Workspace  Design. 

1.  Device  work  spaces  should  be  designed  to  incorporate  easy  and 
flexible  to  technical  documentation.  (6)  p.  31.7 

2.  Students  should  be  able  to  use  the  training  materials  easily 
and  simultaneously  with  the  simulation.  (3)  p.  43.3&4 

E.  Ease  of  use. 

1.  The  operation  of  the  new  training  device  should  not  make  addi¬ 
tional  demands  on  the  students  above  those  of  the  current  training  system. 
(5)  p.  24.1,  (10)  p.  81.1 

2.  The  utilization  of  the  new  training  system  should  be  cost- 

effective  with  reference  to  the  instructor's  time.  (1)  p.  43.2,  (10)  p. 

80.2 


3.  Device  computer  programs  and  user  input/output  features  should 
not  limit  or  distort  instructional  objectives  of  the  system.  (1)  p.  24.9 


4.  Instructor  station  should  be  as  simplified  as  possible  to  as¬ 
sure  ease-of-use,  user  friendliness,  and  minimal  distraction  from  the 
instructor  role.  (1)  p.  43.2,  (3)  p.  35.1 

5.  The  instructor  should  not  require  much  training  to  operate  the 
trainer  [i.e.,  relative  to  the  training  delivered  by  the  training  devices. 
(3)  p.  35.1.  Use  job  aids  to  help  the  instructor,  where  possible,  e.g., 
where  instructor  will  not  be  time  stressed.] 

6.  Touch  panel  input  on  the  CRT  is  preferred  for  student  input. 

(1)  p.  37.5 

7.  The  student  should  be  able  to  respond  easily  and  naturally  to 
the  simulation  and  the  training  materials.  The  student  should  not  require 
special  training  on  idiosyncratic  characteristics  of  the  training  device. 
(3)  43.3 

8.  For  students  with  weak  reading  skills,  the  use  of  touch¬ 
screen  or  use  of  touch-panel  can  simplify  learning  where  procedural 

skills  are  involved.  (9)  p.  86.1 

9.  The  training  device  should  allow  the  student  to  be  trained  on 
tasks  that  cannot  be  trained  on  the  actual  equipment.  (3)  p.  35.3 

10.  Schematics  of  every  device  should  be  provided  to  all  [training 
sites]  (6)  p.  31.1,  (10)  p.  76.1 

11.  HARDWARE 


1.  System  design,  including  cable  length,  will  provide  sufficient 
flexibility  for  ease  of  operation  and  observation.  (1)  p.  27.4 

2.  CRT  will  be  adjustable  to  reduce  glare,  allow  easy  operation  by 
students  and  instructor,  and  allow  the  operator  and  the  display  to  be  ob¬ 
served.  (1)  p.  27.4 

3.  Prototypes  will  be  sufficiently  rugged  to  survive  the  specified 
testing.  (10)  p.  77.1 

4.  Data  back-up  capability  should  be  a  device  feature  where  impor¬ 
tant  student  performance  data  could  be  lost  due  to  loss  of  device  power 
(e.g.,  presence  of  constant  voltage  regulator  or  back-up  battery  power 
system).  (1)  p.  24.11 

5.  Device  features  which  generate  physical  forces  such  as  vibra¬ 
tion  or  fields  should  be  prevented  from  adversely  affecting  training  con¬ 
trols/  stimuli  and  equipment  intended  to  record  student  performance.  (7) 
p.  18.2&4 


6.  Use  video  disks  to  provide  relatively  unchanging  information. 
Store  more  information  on  the  video  disk  than  is  required.  (3)  p.  42.6 


7.  Provide  slide  projector  or  large  magnetic  hard  disks  for  stor¬ 
age  of  changeable  information  and  to  allow  updating/correcting  of  training 
program.  (3)  pp.  33.4,  42.6 

B.  Safety  and  Hazards. 

1.  Device  features  should  be  free  of  physical  dangers  and  irritants 
to  individuals  and  clothing  sudi  as  sharp  corners/edges,  unnecessary  physi¬ 
cal/audio/visual  obstructions,  or  other  hazards  which  interfere  with  student 
and  instructor  performance  or  detract  from  training  conditions.  (1)  p.  27.9, 
51.5 


2.  Cable  connections  will  be  sturdy  and  recessed.  (1)  p.  27.8 

3.  Cables  will  be  sturdy  and  well  protected,  and  equipped  with  ends 
that  prevent  excessive  flexing  motion.  (1)  p.  29.5 

4.  The  simulator  will  be  safer  than  the  actual  equipment  (AE) .  (1) 

p.  36.3 

C.  Maintenance. 

1.  Device  performance  reliability  should  be  high  in  order  to  maintain 
continuity  and  momentum  of  training  sessions  (i.e.,  drill,  motivation,  rein¬ 
forcement).  (7)  p.  19.3 

2.  Training  device  features  should  be  constructed  with  the  goal  of 
95%  operational  availability  [for  system  devices  and  98%  for  non-developmen- 
tal  items]  to  minimize  down  time.  (5)  p.  42.4,  (6)  p.  32.1 

3.  Device  hardware  sudi  as  coils,  bolts,  switches,  etc.,  should  be 
"hardened"  sufficiently  to  withstand  the  rigors  of  student  use.  (7)  p.  20.6 

4.  Any  remove/replace  trainer  should  be  able  to  withstand  the  at- 
tanpts  of  novice  students.  (3)  p.  34.3 

5.  MIL  SPEC  1472C  maintainability  specifications  will  be  required. 
Adequate  access,  illumination,  labels,  wire  coding  etc.  will  be  used. 

Devices  will  be  designed  so  that  major  ccxnponents  may  be  improved  and  re¬ 
placed  easily.  (1)  p.  29.4&5 

6.  A  remove  and  replace  repair  philosophy  will  be  used  to  the  extent 
feasible.  The  device  will  be  designed  so  that  major  components  can  be  eas¬ 
ily  replaced.  Special  test  coo^nents  will  be  provided  as  well  as  replace¬ 
ment  parts.  Suspected  components  will  be  replaced  with  the  test  components 
and  the  systan  tested.  When  the  faults  are  cleared  the  test  components  will 


be  ranoved,  standard  replacement  ccxnponents  inserted,  and  the  system 
retested.  Defective  components  are  repaired  "off  line".  (1)  p.  29.5  |  im¬ 
plied  by  paragraph] 

7.  Equipment  for  operations  training  will  be  designed  so  that  mainte¬ 
nance  can  be  conducted  from  outside  the  device  when  this  will  allow  easier 
access  of  the  components.  (1)  p.  29.5  |  implied  by  paragraph] 


III.  ACQUISITION  MANAGEMENT 

A.  Front  End  Analysis  (FEA) . 

1.  Tasks  to  be  trained  will  be  selected  by  the  initiating  agency. 

(10)  67.1 

2.  Determine  the  requirements  of  the  using  organization  and  incorpo¬ 
rate  then  into  the  SOW.  Also  determine  if  the  requirements  procedures  may  be 
undergoing  review  or  change.  (10)  78.2 

3.  Tasks/training  requirements  for  which  training  devices  are  devel¬ 
oped  should  be  stable.  They  will  be  taught  for  several  years  in  the  same 
MOS.  (10)  67.2. 

4.  The  tasks/specific  training  requirements  will  be  specified  in  the 
SOW.  This  suggests  a  separate  procurement  to  perform  the  FEA.  (10)  62.3, 

71.1 

5.  Specify  front  end  (FEA)  procedures  and  documentation  in  the  SCW. 

(1)  p.  6.4 

6.  More  than  one  subject  matter  expert  will  be  employed  in  FEA  ac¬ 
tivities  requiring  expert  judgement.  (10)  68.1 

7.  The  fidelity  analysis  will  examine  the  range  of  variables  to  be 
encountered  in  the  the  training  environment.  (10)  73.1 

8.  Differences  between  SOP  and  expert  opinion  will  be  found  and  re¬ 
solved.  (10)  69.1 

9.  The  results  of  the  fidelity  analysis  will  be  incorporated  in  the 
device  requiranents.  (10)  72.1 

B.  Evaluation  Requirements 

1.  A  detailed  written  evaluation  plan  is  required.  The  plan  shall  be 
accepted  by  all  involved  government  agencies.  (10)  62.2  [This  should 
detail  a  trainee  performance  test  and  minimal  standards  of  learning  to  in¬ 
clude  tasks,  action,  condition,  and  standard/error  rate  . 


2.  The  internal  formative  training  evaluation  plan  will  be  developed 
as  part  of  the  FEA.  (10)  p.  8.13 

3.  The  external  sunnative  validation  plan  will  be  developed  as  part 
of  the  FEA.  (10)  p.  8.13 

C.  Training  Device  RFP/SOW: 

1.  Terms  used  in  RFP  must  be  adequately  defined.  (10)  p.  27.2 

2.  Precise  device  [functional]  specifications  shall  be  made  available 
to  the  contractors.  (3)  p.  59.4,  (10)  62.1 

3.  Specify  training  device  evaluation  procedures  in  the  SOW. 

(1)  p.  6.2 

4.  The  SOW  must  specify  training  system  products  that  can  be  vali¬ 
dated  (see  rules  on  development  of  validation  plan  prior  to  SOW) .  (10)  p. 

27.3 

5.  The  integration  of  the  training  device  into  the  POI  will  be  speci¬ 
fied  in  the  SOW.  (10)  54.3&4 

D.  Design  and  device  acceptance  procedures; 

1.  Fully  develop  design  acceptance  procedures  and  criteria  before 
contract  award,  preferably  in  the  SOW.  (1)  p.  6.2 

2.  Fully  develop  device  acceptance  procedures  and  criteria  before 
contract  award.  (1)  p.  6.2 

3.  Criteria  for  device  acceptance  shall  be  comprehensive,  precise, 
and  fully  understandable  by  all  parties.  (10)  62.1 

4.  Design  acceptance  procedures  will  specify  how  FEA  incorporation 
into  the  device  will  be  evaluated.  (1)  p.  6.4 

5.  Design  and  device  acceptance  procedures  will  incorporate  training 
criteria  and  procedures  for  determining  whether  they  have  been  satisfied, 
such  as  the  Courseware  Instructional  Features  Checklist.  (1)  p.  6.2 

6.  Apply  relevant  portion  of  MIL-STD-1472C  as  part  of  the  design 

acceptance  procedures.  (1)  pp.  21-29,  53.5 

7.  Apply  MIL-STD-1472C  based  checklist  as  part  of  the  device  accep¬ 
tance  procedures.  (1)  pp.  21-29,  53.5 

8.  Prototype  tests  will  include  tests  which  require  humans  to  exer¬ 
cise  all  features  of  the  device.  Problems  will  be  noted  and  corrected.  (1) 
p.  8.1 


9.  Transfer  of  training  studies  will  be  conducted  on  the  prototype 
device.  (1)  pp.  8.1 

10.  Post-contract  award  changes  to  the  training  device  will  be  re¬ 
flected  in  changes  to  the  design  and  device  acceptance  procedures.  (1)  p. 

6.2 

11.  In  addition,  design  and  device  acceptance  procedures  will  include 
the  following  items: 

(a)  Device  requires  differential  students'  i.e.,  discrimination 
and  generalization  learning  responses. 

(b)  Device  provides  unambiguous,  concise  feedback  to  response. 

(c)  Material  is  presented  in  order  of  difficulty  or  according 
to  the  logic  of  the  discipline. 

(d)  Mastery  of  current  material  is  required  before  student 
progresses  to  more  difficult  material. 

(e)  Previously  mastered  material  is  tested  for  retention  later 
in  the  course  ((1)  p.  51.1] 

(f)  The  assignment  of  functions  to  the  trainee,  the  instructor, 
the  device,  and  "other"  must  be  presented  formally  for 
design  acceptance  and  then  checked  during  the  device 
acceptance  phase.  (1)  pp.  52.5-53.1  |implied| 

12.  Reliability  and  maintainability  (RAM)  analyses  will  be  conducted. 
(10)  74.1 

13.  Full  documentation,  including  schematics,  will  be  provided  with 
the  equipment.  (10)  76.1 

14.  Determine  that  the  projected  user  can  fully  support  the  projected 
system.  (10)  80.1 

15.  Student  performance  records  will  be  produced  and  stored  by  all 
automated  training  devices.  The  records  will  provide  the  student's  name  and 
ID#.  They  will  identify  the  training  attempted  and  passed  as  well  as  time 
elapsed,  error  scores  and  correct  steps  per  minute  for  each  segment.  (1)  p. 
43.5-44.3,  53.3. 

16.  Automated  training  materials  will  be  easily  updated  and  modified 
by  the  instructors.  (1)  p.  43.5-44.3,  53.3. 

17.  Instructors  will  be  trained  in  all  phases  of  the  training  de¬ 

vices.  They  will  be  trained  and  provided  with  materials  required  to  train 
other  instructors.  (3)  p.  60.5-6,  50.8 


IV.  HUMAN  FACTORS  and  SOFTWARE  DESIGN 


A.  Instructor  Station. 

1.  The  instructor  station  will  be  designed  for  an  instructor,  not  a 
computer  expert.  (1)  p.  37.9 

2.  Source  code,  compiler  and  linker  programs  will  be  provided  in 
hard  copy  and  in  computer  accessible  form.  (1)  p.  37.7,  (3)  p.  35.2 

3.  The  computer  programming  shall  be  done  in  a  high  level  language. 
The  language  chosen  will  be  one  that  can  be  mastered  sufficiently  by  either 
the  training  developers  or  instructors  that  they  can  make  the  desired 
changes.  [(1)  p.  37.7,  (3)  p.  35.2]  The  changes  include,  but  are  not  limited 
to: 


reordering  and  skipping  lessons  [(6)  p.  32.2]; 

updating  and  modifying  materials  [(1)  p.  36.2&3,  p.  43.2]; 

constructing  new  lessons  [(1)  p.  43.2]. 

4.  Instructional  authoring  languages  should  be  considered  which  will 
allow  the  training  developer  or  instructor  to  change  the  instructional  mate¬ 
rial  easily  and  without  extensive  training.  (3)  p.  34.2 

B.  Student  Station. 

1.  Device  software,  including  authoring  systems  should  provide 
in-depth  on-line  user  guideince/assistance/help  to  the  students,  instructors 
and  maintainers  [(1)  p.  29.1&6;  (6)  p.  32.4]  including  on-line  cissistance  in 
system  set  up.  (3)  p.  33.3 

2.  Lessons  may  be  restarted  without  stopping  the  system  or  losing 
existing  data.  They  may  be  restarted  where  they  stopped  rather  than  at  the 
beginning.  Re-booting  is  viewed  by  students  and  instructors  as  a  poor  method 
for  restarting  a  lesson  or  instructional  module  and  should  be  avoided  in 
device  software  design.  (1)  p.  36.2;  (7)  p.  18.12 

3.  Computer  system  messages  will  be  clear,  concise  and  uncoded.  (1) 

p.  29.1&6 


4.  Computer-based  systems  will  indicate  their  status  continuously. 
The  status  indications  will  be  ready  for  input,  computing  normally,  error 
status.  (1)  p.  27.4,  p.  29.1,  p.  50.3 

5.  Computer-based  systems  will  use  error  trapping  routines  to  re¬ 
cover  from  unanticipated  student  and  equipment  errors.  A  default  error 
trapping  routine  will  indicate  an  error  when  the  internal  clock  indicates 
the  current  task  has  exceeded  a  preset  value.  (1)  p.  27.4 


V.  INSTRUCTIONAL  DELIVERY 


A.  Instructional  Features  and  Principles. 

1.  Determine  mean  and  range  of  entering  students'  knowledge  and 
skills.  (10)  p.  83.1 

2.  Viability  of  the  device  as  an  instructional  delivery  system 
should  not  be  based  only  upon  transfer-of-training  hypotheses  but  also  upon 
the  considerations; 

(a)  Cost, 

(b)  Maintenance  requirements, 

(c)  Instructional  flexibility, 

(d)  Ease  of  use  (6)  p.  31.8 

3.  Instructional  flexibility  of  a  training  device  is  an  attribute 
that  should  be  sought  in  device  design  and  can  be  assessed  by  the  following 
variables  (there  may  be  additional  criteria  as  well): 

(a)  Training  can  be  altered  at  low  cost  and  in  little  time; 
((7)  p.20.2] 

(b)  Student  progress  can  be  self-paced;  ((7)  p.  20.2,  (10) 
p.  44.5] 

(c)  The  order  of  training  and  segments  can  be  readily 
changed;  [(7)  p.20.2j 

4.  Device  features  design  should  place  enphasis  upon  providing  opp¬ 
ortunity  for  hands-on  student  practice  with  the  training  device.  (6)  p. 

31.6 

5.  Device  features  design  should  reflect  principles  of  good 
instructional  practices.  Courseware,  for  example,  should  not  submit  to 
hardware  limitations  when  the  net  result  is  diminished  instructional  effe¬ 
ctiveness.  (1)  p.  52.3 

6.  Device  features  which  provide  training  stimuli  should  limit  the 
number  of  stimuli  presented  at  any  one  time  to  avoid  requiring  inordinate 
attention  from  students  (i.e.,  should  support  focused  instruction).  (7)  p. 
20.5 

7.  To  the  extent  possible  device  features  should  carry  out  training 
independent  of  the  instructor  so  that  the  instructor  is  free  to  monitor  stu¬ 
dent  performance  and  provide  student  guidance/help.  (1)  p.  52.4 

8.  Training  features  of  the  device  should  accommodate  various  readi¬ 
ness  levels  of  students  so  that  the  students  can  engage  in  a  device  -trained 
task  at  a  level  their  skills  accommodate.  (10)  p.  83.1 


9.  Learned  behaviors  should  be  applied  immediately.  (10)  p.  44.3 


10.  Random  malfunction  selection  is  a  good  feature,  but  must  not  be 
the  only  way  to  select  malfunctions.  (3)  p.  34.4 

11.  Maintenance  trainers  should  provide  a  "troubleshoot  only"  mode. 
Part  task  trainers  for  "troubleshooting  only"  should  be  considered.  (3)  p. 
34.6 


12.  The  simulator  will  be  designed  to  teach  the  hazards  of  the  AE 

better  than  actual  equipment.  (1)  36.1,  51.5 

13.  Students  should  not  be  allowed  access  to  "help"  until  they  have 
tried  the  task.  (3)  p.  43.5 

14.  Students  who  have  requested  "help"  should  be  retested  on  that 
module  later  in  the  training.  (10)  p.  83.3 

15.  Material  will  be  presented  in  order  of  difficulty  [(1)  p.  51.1, 
(10)  p.  44.4]  or  according  to  the  logic  of  the  discipline  (1)  p.  51.1 

16.  The  training  device  will  require  differential  student  responses. 
(1)  p.  39.1 

17.  Performance  feedback  should  be  provided  to  the  trainee.  It 
should  be  unambiguous,  concise  and  diagnostic,  telling  the  student  how  to 
improve.  (3)  p.  34.5,  (1)  p.  51.1 

18.  Mastery  of  current  material  is  required  before  student  progresses 
to  more  difficult  material.  (1)  p.  51.1 

19.  Previously  mastered  material  is  tested  for  retention  later  in  the 
course.  (1)  p.  51.1 


B.  Student  record  keeping. 

1.  The  instructor  CRT  can  display  both  summary,  and  detailed  infor¬ 
mation  on  the  progress  of  each  student  and  the  class  while  the  student  is 
working.  (1)  p.  21.2 

2.  Student  performance  records  will  be  produced  and  stored  by  all 
automated  training  devices.  The  records  will  provide  the  student's  name  and 
ID#.  It  will  identify  the  training  attempted  and  passed  as  well  as  time 
elapsed,  error  scores  and  correct  steps  per  minute  for  eadi  segment.  (1)  p. 
43.5 

3.  Student  performance  records  should  be  based  on  FEA  task  analysis. 
(3)  p.  32.6 
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4.  Student  performance  records  will  be  self-explanatory  to  the  stu¬ 
dent  and  the  instructor.  The  output  will  be  properly  labeled  and  will  not 
require  a  code  book  for  interpretation.  They  will  be  available  on  line  dur¬ 
ing  the  instructional  sessions  and  will  also  be  available  in  suninary  format 
across  both  training  segments  and  students.  (1)  p.  43.5-44.1,  48.3,  50.3&4 


5.  The  software  that  records  student  performance  should  provide  the 
instructor  with  the  ability  to  correct  and  annotate  the  record  to  identify 
equipment  failures  and  other  exceptional  events,  thereby  preventing  mis¬ 
information  about  student  performance  and  the  information  required  for  re¬ 
pair.  (1)  p.  50.4,  (3)  p.  33.1 


VI.  FIDELITY 


A.  Basic  considerations. 


1.  Determining  the  level  of  fidelity  for  device  features  should 
take  into  account,  at  least  generally: 


(a)  Task  type, 

(b)  Task  difficulty, 

(c)  Specific  skills  required  to  performed  the  task, 

(d)  Trainee  sophistication, 

(e)  Stage  of  training, 

(f)  Training  context, 

(g)  Device  role  in  the  overall  POI, 

(h)  User  acceptance  of  the  device  , 

(i)  Supportive  instructional  features  (10)  p.  73.2 


2.  Simulation  devices  will  not  blindly  replicate  the  actual  equip¬ 
ment  if  improved  training  or  maintenance  can  be  provided.  (1)  52.3 


3.  Emphasize  development  of  part-task  and  lower  fidelity  maintenance 
simulators  over  major  overall  simulators.  (10)  p.  12.1 


B.  Two  vs.  Three  Dimensional  Considerations. 


1.  A  moderate  level  of  physical  fidelity  in  a  training  device  may  be 
adequate  for  training  a  troubleshooting  task  (e.g.,  computer  graf^iics  or 
videodisc  could  be  an  adequate  replacement  for  3-D  hardware) .  For  remove  and 
replace  tasks,  however,  higher  fidelity  3-D  simulation  is  preferable.  (6)  p. 
31.9 


2.  Most  automotive  and  electronic  maintenance  tasks  can  be  taught 
with  2-D  media,  and  3-D  high  fidelity  is  not  required.  (10)  p.  53.2,  (1)  p. 
37.3  Later  studies  have  shown  that  2-D  is  effective  for  teaching 
trouble-shooting  to  relatively  advanced  trainees,  but  that  AE  is  better  for 
remove/  replace  level  training. 
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procedures.  (3)  p.  32.3 

3-D  training  media  must  be  thoroughly  checked  to  determine  if 
>t-effective.  (10)  p.  53.2 


itudents.  Advanced  students  often  profit  from  reading  and 
(3)  p.  56.1 
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VII.  NOTES 

1.  Reading  level  of  text  should  match  that  of  students.  Students' 
reading  grade  level  may  be  estimated  from  their  AFQT  scores.  (As  a  design 
figure,  the  reading  grade  level  should  correspond  to  the  AFQT  entrance 
requirement  associated  with  the  MOS.)  The  reading  grade  level  of  text  may 

be  assessed  by  the  FORCAST  formula.  (As  a  design  specification,  a  toler-ance 
of  plus  or  minus  one-half  a  level  should  be  used.  (1)  p.  26. 

2.  The  "mouse"  is  the  preferred  means  of  cursor  moving  for  long  data 
entry  sessions  because  it  avoids  the  arm  fatigue  experienced  in  pointing  a 
light  pen  for  long  periods  of  time.  (2)  p.  344 

3.  The  "highlight- response"  system  is  the  preferred  means  of  select¬ 
ing  an  item  from  a  menu.  (2)  p.  346. 

4.  When  using  a  light  pen  dialog,  make  certain  that  the  "targets"  for 
the  light  pen  are  at  least  1/4"  square.  (3)  p.  1.1-14. 
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Kern,  R.  P.,  Sticht,  T.  G.,  Welty,  D. ,  &  Hauke,  R.  N.  (1976).  Guidebook  for 
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Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences.  (AD  A033  935) 

Kantowitz,  B.  H.,  &  Sorkin,  R.  D.  (1983).  Human  factors.  New  York:  Wiley. 


Sidorsky,  R.  C.,  Parrish,  R.  N.,  Gates,  J.  L.,  &  Hunger,  S.  J.  (1984).  De¬ 
sign  guidelines  for  user  transactions  with  battlefield  automated  systems; 
Prototype  for  a  handbook.  Research  Product  84-08.  Alexandria,  VA:  U.S. 
Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 

(AD  A153  231 ) 
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APPENDIX  B. 

REVERSE  ENGINEERING 

The  DAG  is  a  conpilation  of  lessons  learned  about  training  device  acqui¬ 
sition  during  the  course  of  the  A^f^ESS  program.  The  lessons  were  learned  the 
hard  way,  i.e.,  when  the  devices  were  put  to  use,  shortcomings  becane  appar¬ 
ent.  Because  AMTESS  was  a  research  program,  the  shortcomings  and  their 
causes  received  considerable  documentation.  Are  the  resulting  lessons 
learned  applicable  to  current  training  device  acquisitions?  To  answer  this 
question  the  reverse  engineering  exercise  analyzed  two  specification  docu¬ 
ments  developed  to  acquire  training  devices  for  the  Bradley  Fighting  Vehicle. 

The  analysis  procedure  was  straightforward.  Research  psychologists 
familiar  with  the  DAG  searched  specifications  for  "Bradley  Fighting  Vehi¬ 
cle  Panel  Trainers"  and  "BFVS  HOT"  to  see  if  the  items  of  the  DAG  were 
addressed.  Three  categories  of  data  were  enumerated:  "treated",  "not 
treated"  and  "violated".  "Treated"  meant  that  a  phrase  or  paragraph  in 
the  specification  addressed  the  concern  of  the  DAG  item.  Thoroughness  or 
rigor  were  not  judged.  "Not  treated"  meant  there  was  no  indication  that 
the  DAG  item  had  been  considered  in  the  specification;  "violated"  meant 
that  the  specification  expressed  a  requirement  counter  to  that  of  the  DAG 
item.  Summary  results  are  shown  in  Table  1. 


Table  1 


Appearance  of  DAG  Items  in  Specifications 
for  IWo  Bradley  Training  Devices 

Training  Device 


BFV  Panel  Trainers  BFV  HOT 

No.  of  items  treated  32  30 
No.  of  items  not  treated  99  99 
No.  of  items  violated  7  9 


Total  no.  of  items  138  138 


Tables  2  and  3  show  breakdowns  by  DAG  category  for  the  two  devices. 


Table  2 

Appearance  of  DAG  Items  in  Specification  for 
Bradley  Fighting  Vehicle  Hands-On  Trainer 
by  DAG  Category 

Number  of  Items 

DAG  Category  Treated  Not  Treated  Violated 


Human  Factors  (34  items) 

5 

27 

2 

Hardware  (19) 

10 

9 

0 

Acquisition  Managenent  (39) 

9 

26 

4 

Human  Factors  and 

Software  Design  (12) 

2 

10 

0 

Instructional  Delivery  (26) 

3 

22 

1 

Fidelity  (8) 

1 

5 

2 

Table  3 


i^pearance  of  DAG  Itenns  in  Specification  for 
Bradley  Fighting  Vehicle  Panel  Trainers 
by  DAG  Category 


Number  of  Items 


DAG  Category 

Treated 

Not  Treated 

Violated 

Human  Factors  (34  items) 

8 

26 

0 

Hardware  (19) 

10 

9 

0 

Acquisition  Managanent  (39) 

8 

27 

4 

Human  Factors  and 

Software  Design  (12) 

2 

10 

0 

Instructional  Delivery  (26) 

4 

21 

1 

Fidelity  (8) 

0 

6 

2 

32 

99 

7 

Examples  of  itans: 

The  following  are  examples  of  DAG  items  and  how  they  were  classified. 
Not  Treated 

"A  signal  (audio  or  visual)  will  not  have  multiple  meanings." 

"Touch  panel  input  on  the  CRT  is  preferred  for  student  input." 

"Student  performance  recoerds  will  be  self-explanatory  to  the  student 
and  the  instructor." 

"Previously  mastered  material  is  tested  for  retention  later  in  the 
course." 

Violated 

"The  SOW  must  specify  training  system  products  that  can  be  vali¬ 
dated. 
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"To  the  extent  possible  device  features  should  carry  out  train¬ 
ing  independent  of  the  instructor  so  that  the  instructor  is  free  to 
monitor  student  performance  and  provide  student  guidance  and  help." 

"Prototype  tests  will  include  tests  which  require  humans  to  exercise 
all  features  of  the  device." 


Treated 


"System  design,  including  cable  length,  will  provide  sufficient 
flexibility  for  ease  of  operation  and  observation." 

"Cable  connections  will  be  sturdy  and  recessed." 

"MIL  SPEC  1472C  maintainability  specifications  will  be  required." 

"Reliability  and  maintainability  (RAM)  analyses  will  be  conducted." 
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